Advice-Taking Metaanalysis

Summer 2025 Report

Overview

I have been working on converting the Himmelstein (2022) advice-taking dual-hurdle model from a two-level cross-classified model, with responses cross-classified within judges and items, into a three-level cross-classified model, with responses cross-classified within judges and items and then nested within studies. To do this, I added a third level to the multilevel structure of the model that would account for study-level variance in the intercept estimates in both the part of the model that estimates the probabilities of Decline, Adopt, and Compromise (DAC) as well as the part that estimates the continuous Weight of Advice (WOA) values by modifying the Stan code and implementation in R. I then fit the model to the full dataset of WOA studies.

Results

Displayed below is a plot of the distribution of WOA values as per the data and as per the model predictions. We can see the large proportion of WOA values near zero and one, with a range of values between as the DAC framework would expect.

One noticeable piece here is the bump at .5 exactly, with lower density surrounding it. This seems to be a study-level phenomenon—some studies have many responses with WOA values of exactly .5. Below are the studies with the highest proportions of the .5 responses.

The average proportion of .5’s in an given study was 5.24%.

It may be helpful to identify why certain studies have more responses of exactly .5 than others—the model seems to be fitting the data well across all values except the center.

Below we can see the distribution of the study-level intercepts for the probability of responses falling into the DAC categories. To the right we can see those box plots representing the distribution of those sampled intercepts for each study.

Next step is to decide how to handle these high .5 studies and to add predictors to the model! Looking forward to hearing what predictors would be best to include in this analysis.